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ABSTRACT 


Test  and  evaluation  (T&E)  managers  often  control  testing  via  heuristics  (i.e., 
using  experience  and  lessons  learned  from  previous  testing  to  modify  existing  range 
control  procedures).  An  exclusively  heuristic  approach  may  prove  difficult  in  predicting 
test  control  issues  with  new  systems.  Given  this  limitation,  this  thesis  poses  the  following 
question:  Can  formally  modeling  the  process  of  conducting  test  range  events  expose 
previously  overlooked  ambiguities  and  identify  high-value  decision  points? 

This  thesis  explores  how  formalization  of  these  experience-based  decisions  as  a 
process  model  representing  a  T&E  event  may  reveal  high-value  decision  nodes  where 
certain  decisions  carry  more  weight  or  potential  for  impacts  to  a  successful  test.  The 
thesis  evaluates  behavioral  modeling  techniques  and  language,  ultimately  using  the 
Innoslate  modeling  software  tool  to  construct  a  formal  model  of  the  range  decision 
process.  The  thesis  presents  the  results  of  the  simulation  runs  in  Innoslate  and  shows  how 
having  the  formal  model  improves  on  the  simplified  model  by  expanding  the  process 
from  two  to  eleven  decision  points.  The  thesis  concludes  that  the  formal  model  has  use  as 
a  planning  tool  that  can  assist  mangers  in  anticipating  problems  and  focusing  resources 
on  resolving  these  issues. 
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EXECUTIVE  SUMMARY 


Acoustic  range  managers  need  a  better  system  or  process  for  identifying  high- 
value  decision  points  before  conducting  test  events.  At  the  time  of  this  research,  a 
qualitative  process  model  that  represents  the  acoustic  range  decision  process  did  not 
exist.  This  thesis  reviewed  decision-modeling  literature  to  evaluate  possible  models  and 
applications  for  representing  the  decision  processes  of  an  acoustic  range  test  event. 

The  research  focused  on  modeling  the  decision  process  of  the  Naval  Undersea 
Warfare  Center  (NUWC)  Division,  Keyport,  underwater  acoustic  tracking  range.  The 
model  had  to  be  capable  of  replicating  a  simplified  version  of  an  event,  so  test  planners 
can  see  the  identification  of  high-value  decision  nodes  and  use  this  information  to  make 
better  decisions  regarding  risk  management  and  mitigation.  Research  focused  on 
developing  a  basic  understanding  of  the  decision  methods,  developing  an  appropriate 
model,  and  then  simulating  the  decision  process  to  identify  high-value  decision  points. 

Range  managers  relied  on  heuristics  to  predict  problem  areas,  which  do  not 
always  provide  useful  risk  analysis  tools  for  new  testing  programs  that  may  require 
different  evaluation  constructs.  Additionally,  the  range  did  not  employ  quantitative  risk 
analysis  techniques,  which  could  have  identified  problem  areas  before  testing.  The  author 
focused  on  providing  a  formal  model  that  could  provide  these  capabilities  and  addressed 
the  following  question:  Can  formally  modeling  the  business  process  of  conducting  test 
range  events  expose  previously  overlooked  ambiguities  and  identify  high-value  decision 
points? 

Next,  the  thesis  assessed  a  proposed  formalized  decision  process  capable  of 
representing  a  simplified  range  event  and  used  the  following  measures  of  merit  to 
evaluate  the  effectiveness  of  the  formal  model: 

•  The  number  of  high-value  decision  points  reflected  in  the  formalized 
process  model  exceeds  the  number  of  high-value  decision  points  in  current 
informal  process  models  of  range  event  activity.  High-value  decision 
points  reflect  specific  places  in  the  test  process  that  if  improperly 
executed,  could  result  in  loss  of  test  data  or  prematurely  aborting  the  test. 
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•  The  formal  model  displayed  at  a  minimum  the  decision  points  found  in  the 
simplified  model.  The  formal  model  improved  on  the  simplified  model  by 
providing  more  decision  points,  which  represented  the  test  event  more 
accurately. 

Model  variables  included  the  number  of  decision  nodes  and  the  number  of  uses  of 
a  decision  node  during  the  simulation  run.  Multiple  uses  of  the  same  decision  node 
indicated  a  high-value  decision  point.  Confirmation  of  high-value  decision  nodes 
occurred  during  model  validation.  The  following  two  points  summarize  the  risks  and 
uncertainties: 

•  Since  the  acoustic  range  studied  had  never  modeled  their  decision  process, 
the  ability  to  replicate  the  informal  process  as  a  formal  model  via  a 
simulation  program  was  uncertain.  Ultimately,  the  thesis  identified  a 
suitable  program  that  adequately  represented  the  range  decision  process. 

•  Similarly,  the  ability  of  the  formal  model  to  identify  high-value  decision 
points  in  the  test  decision  process  was  unknown.  In  the  course  of  building 
the  formal  model,  methods  provided  by  the  software  program  allowed 
counting  decision  node  usage,  which  in  turn  provided  information  on  the 
value  of  each  decision  in  the  process  model. 

Current  business  process  modeling  (BPM)  methods  were  evaluated.  The  author 
evaluated  the  simulation  of  the  range  process  model  using  the  selected  process  model 
software.  Aspects  observed  included  logical  flow  of  events,  event  duration,  decision 
probabilities,  number  of  uses  of  each  decision  node,  accuracy  of  feedback  loops,  and 
overall  utility  of  the  model  to  range  managers.  Additionally,  the  author  shared  the  model 
simulation  and  results  with  range  managers  and  received  feedback  that  the  range  could 
use  the  product  when  conducting  event  planning,  and  risk  assessments. 

The  thesis  evaluated  the  current  modeling  approaches  used  by  the  acoustic  range 
through  the  collection  of  relevant  decision  and  process  data  used  to  control  range  events. 
The  main  sources  were  existing  range  control  policy  documents  and  information 
collected  from  range  managers. 

The  thesis  documented  the  development  of  a  simplified  block-diagram  model 
representing  a  routine  acoustic  range  event.  The  thesis  developed  a  formal  process  model 
using  simulation  software  capable  of  replicating  the  simplified  process  model.  Conduct  of 
multiple  model  runs  determined  that  the  model  could  identify  high-value  decision  nodes. 


Observation  of  modeling  runs  assisted  in  determining  whether  required  the  formal 
model  adequately  replicated  feedback  loops  and  processes  performed  the  range  operators. 
Model  runs  showed  the  actual  sequence  of  expected  high-value  decision  points  in  the  test 
event  process.  The  simulation  software  generated  reports  providing  data  on  the  number  of 
uses  of  each  decision  point  during  a  simulation  run. 

At  the  conclusion,  the  thesis  defined  that  a  high  number  of  uses  of  the  same 
decision  point  throughout  the  modeling  runs  indicated  a  high  value,  as  the  range 
operators  would  have  multiple  opportunities  to  make  the  decision.  Therefore,  the  higher 
number  of  times  a  range  operator  encountered  that  decision,  the  higher  the  probability 
that  an  incorrect  choice  at  that  point  could  negatively  affect  the  test  run.  The  model  had 
sufficient  discrimination  between  decisions  to  provide  clear  indications  as  to  which 
decisions  points  had  high  value  according  to  the  definition. 

The  capability  of  the  model  to  adjust  decision  choice  probabilities  provided  the 
ability  to  observe  changes  in  the  number  of  uses  of  each  decision,  indicating  the  model 
had  sufficient  sensitivity  to  show  how  the  accuracy  of  choice  probabilities  affect 
identification  of  high-value  decisions.  Test  and  evaluation  subject  matter  experts  and 
managers  from  the  Naval  Undersea  Warfare  Center  Division,  Keyport,  reviewed  the 
model  execution  and  modeling  run  results.  They  concluded  the  model  has  application  to 
range  test  event  planning  and  with  further  refinement,  could  assist  test  planners  and  range 
managers  in  predicting  potential  risk  areas  during  range  events.  Additionally,  the  model 
could  assist  managers  in  assessing  new  test  programs  prior  to  their  first  test  runs, 
providing  advance  insight  as  to  where  problems  might  occur. 
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I.  INTRODUCTION 


The  purpose  of  this  chapter  is  to  present  problems  faced  by  undersea  test  ranges  in 
the  area  of  human  decision-making  and  the  lack  of  pre-test  quantitative  analysis  tools 
available  to  assist  range  operators  in  identifying  and  reducing  decision  risks.  This  chapter 
identifies  problem  areas  and  their  effect  on  the  conduct  and  control  of  acoustic  test  events 
and  addresses  why  improvements  in  these  areas  would  benefit  range  managers  and 
operators.  Additionally,  this  chapter  presents  the  thesis  research  questions  and  research 
methodology.  The  expected  product  of  the  research  is  discussed,  specifically,  the 
development  of  a  formal  model  that  simulates  the  decision-making  process  of  a  test  and 
evaluation  (T&E)  event  on  an  underwater  tracking  range.  The  chapter  also  explores  the 
need  for  a  repeatable  method  that  can  formalize  simplified  decision  model  process  charts. 

A.  PROBLEM  BACKGROUND 

Underwater  acoustic  test  and  evaluation  ranges  support  developmental  test  and 
evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E)  necessary  for  undersea 
weapons  and  systems  development.  Control  of  range  operations  includes  procedures  to 
regulate  range  operations  through  systematic  control  of  activities  such  as  the  loading  and 
launching  of  high-speed  test  vehicles.  These  range-event  operations  also  include  free- 
play  approaches  such  as  providing  a  controlled  test  area  for  autonomous  unmanned 
underwater  vehicle  (UUV)  operations.  According  to  Brian  Adams,  range  manager  for  the 
acoustic  range  at  NUWC  Division,  Keyport,  planning  for  these  events  has  historically 
followed  a  heuristic  approach,  (i.e.,  lessons  learned  over  years  of  operating  the  range 
influence  the  planning  and  conduct  of  subsequent  tests).  Event  planners  use  this 
experience  to  develop  test  plans  that  will  accomplish  program  objectives  while 
minimizing  risks  to  the  units  under  test  (UUT),  personnel,  and  range  equipment  (Brian 
Adams,  pers.  comm.  November  20,  2016).  This  method  has  been  sufficient  for  test 
control  of  well-understood  systems  such  as  current  torpedoes  and  submarines.  However, 
this  approach  may  not  be  adequate  to  provide  acceptable  risk  management  for 
increasingly  sophisticated  systems  such  as  autonomous  UUVs  and  new  generation 
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undersea  weapons.  Bodmer  (2003)  discussed  the  need  for  test  and  evaluation 
methodology  to  remain  transformational,  and  that  testing  must  be  as  early  in  the  life  cycle 
as  possible  to  lower  cost.  The  current  range  event  planning  approach  may  not  be 
sufficient  to  support  Bodmer’s  recommendation.  Discussion  with  the  NUWC  Division, 
Keyport  Range  Manager  identified  the  following  shortfalls  in  the  current  heuristic  test 
planning  approach: 

•  lack  of  pre-event  analysis  tools 

•  lack  of  quantitative  risk  assessment  methods 

•  unavailability  of  predictive  models 

•  in  situ  decision-making  limitations 

•  increased  cost  of  poor  decisions 

1.  Lack  of  Pre-test  Event  Analysis  Tools 

At  present,  event  planners  have  no  analysis  tools  that  assist  them  in  predicting 
critical  points  during  the  test  event  when  key  decisions  may  be  likely.  Planning  for  tests 
currently  involves  the  development  of  a  run  plan  (RP)  that  sets  the  sequence  of  events 
during  the  test,  provides  parameters  for  sensors  and  test  equipment,  and  provides  overall 
instructions  to  range  operators  supporting  the  test.  Planners  use  qualitative  risk  analysis 
tools  when  developing  the  RP  and  mitigate  risk  by  following  best  practices  learned  from 
previous  tests,  according  to  the  NUWC  Division,  Keyport  Range  Manager.  When 
encountering  a  problem  during  testing,  range  managers  conduct  inquiries  into  the  root 
causes  of  the  problem  and  then  develop  new  policies,  processes,  or  procedures  designed 
to  prevent  recurrence  of  the  issue.  Managers  apply  these  changes  to  the  RP  process  if 
warranted.  The  NUWC  Division,  Keyport  Range  Manager  also  stated  that  at  present,  the 
range  has  no  prescribed  procedures  for  developing  test  plans  other  than  using  this 
standardized  run  plan  format. 

2.  Lack  of  Quantitative  Risk  Assessment  Methods 

Risk  assessments  tend  to  be  qualitative  and  focus  on  procedural  compliance  and 
centralized  control  of  the  test  event  by  one  individual,  the  range  officer  (RO).  According 
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to  NUWC  Division,  Keyport  Range  Manager,  the  ability  of  the  RO  to  keep  track  of  test 
parameters  limits  the  speed  of  execution  and  maximum  scope  of  the  test  event.  He  further 
stated  not  all  RO  candidates  display  the  necessary  aptitude  to  control  highly  complex 
tests. 

According  to  the  NUWC  Division,  Keyport  Range  Manager,  test  control  currently 
relies  on  a  countdown-like  procedure  with  periodic  stopping  points  that  provide 
intentional  delays  during  the  test  event,  allowing  the  RO  to  review  preparations  and 
evaluate  readiness  to  move  to  the  next  phase.  The  RO  authorizes  the  commencement  of 
each  phase  of  the  procedure,  and  does  not  start  the  next  phase  until  all  steps  of  the  current 
phase  are  complete.  Most  tests  follow  a  regimented  procedure  to  assist  the  RO  in 
controlling  the  test.  Historically,  range  operators  have  not  quantitatively  identified  or 
catalogued  high-value  decision  points  that  occur  during  acoustic  test  events.  High-value 
decision  points  are  those  situations  during  the  test  event  in  which  a  poor  decision  could 
result  in  the  unnecessary  aborting  of  the  test,  loss  of  test  data,  damage  to  the  unit  under 
test  (UUT),  or  damage  range  testing  assets.  He  also  indicated  that  the  lack  of  any  pre-test 
event  decision  analysis  requires  that  ROs  make  decisions  in  situ  using  experience  and 
heuristic -based  procedures. 

3.  Unavailability  of  Predictive  Models 

The  acoustic  range  does  not  use  modeling  or  statistical  analysis  of  decision 
processes  in  its  test  planning,  nor  is  any  quantitative  risk  analysis  performed.  The  testing 
of  torpedo  software  uses  simulation,  but  this  analysis  is  independent  of  range  operations. 
According  to  the  NUWC  Division,  Keyport  Range  Manager,  predictive  modeling  of 
decision  processes  or  effectiveness  has  not  been  evaluated  using  statistical  techniques. 
Analysis  of  completed  test  events  to  identify  decision  points  is  untried.  He  noted  his 
range  managers  are  unaware  if  such  approaches  would  be  of  benefit. 

4.  In  Situ  Decision  Limits 

The  NUWC  Division,  Keyport  Range  Manager  noted  that  Range  Officer  training 

focuses  on  recognizing  and  understanding  normal  and  expected  conditions  for  the 

particular  test  and  on  using  information  from  displays  and  reports  from  other  range 
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technicians  to  determine  if  the  test  is  proceeding  as  planned.  If  the  test  run  deviates  from 
the  plan,  the  range  officer  must  make  the  decision  either  to  continue  the  test  as  planned, 
to  modify  the  plan,  or  to  abort  the  test.  RO  training  requires  aborting  of  tests  only  if  it 
becomes  clear  the  vehicle  or  range  personnel  will  become  at  risk;  otherwise,  the  RO  will 
attempt  to  alter  test  parameters  to  achieve  as  many  test  objectives  as  possible.  He  also 
stated  a  major  purpose  of  the  RO  training  process  is  to  assist  the  RO  in  recognizing 
abnormal  conditions  that  require  intervention,  but  that  ROs  develop  this  skill  more 
thoroughly  through  actual  experience  gained  after  qualification. 

5.  Increased  Cost  of  Poor  Decisions  That  Result  in  Loss  of  Test  Data 

Test  and  Evaluation  events  that  do  not  produce  the  data  required  by  the  program 
can  result  in  delays  to  development  schedule  or  increased  cost  of  testing  if  critical  data  is 
still  required.  Developmental  test  and  evaluation  (DT&E)  events  are  usually  earlier  in  the 
program  cycle  and  often  have  a  limited  budget  (AcqNotes  2016).  The  NUWC  Division, 
Keyport  Range  Manager  stated  acoustic  range  events  are  expensive  given  the  cost  of  test 
vehicles,  range  craft,  and  the  personnel  labor  required  for  planning  and  conduct  of  on- 
water  tests.  Additionally,  reductions  in  DT&E  and  OT&E  budgets  have  forced  program 
managers  to  reduce  testing,  which,  in  turn,  increases  the  value  of  remaining  tests  in  terms 
of  achieving  program  test  objectives.  As  the  purpose  of  DT&E  testing  is  to  decrease 
overall  cost  through  early  discovery  of  failures  and  defects  (Test  Resource  Management 
Council  2017),  acoustic  range  test  failures  can  add  significant  cost  and  schedule  delays  to 
programs. 

These  factors  place  increased  burden  on  the  test  facility  to  conduct  accurate  and 
successful  tests  that  produce  the  intended  test  conditions,  which  in  turn  raises  the  pressure 
on  test  planners  to  predict  and  mitigate  potential  test  problems.  Additionally,  there  is 
increased  burden  on  ROs  to  identify  quickly  emergent  problems  during  testing  and  take 
prompt  action  to  salvage  test  objectives  and  protect  program  test  assets.  If  these  efforts 
fail,  programs  can  lose  critical  data,  which  can  result  in  repeating  an  expensive  test  event. 
Therefore,  poor  decisions  by  range  planners  and  operators  can  severely  affect  programs 
by  wasting  scarce  schedule  and  funding  resources. 
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B.  PROBLEM  FORMULATION  AND  RESEARCH  OBJECTIVES 

The  problem  faced  by  Range  Managers  and  test  planners  centers  on  the  scarcity 
of  predictive  tools  that  allows  range  personnel  to  anticipate  high-value  decisions  in  their 
test  process.  The  high  cost  of  DT&E  and  OT&E  testing  means  programs  often  have 
diminishing  resources  to  conduct  tests,  therefore  each  test  has  increasing  value  in 
producing  useful  data.  Range  managers  require  a  method  capable  of  analyzing  current 
test  events  for  improvements  as  well  as  predicting  potential  issues  for  T&E  of  new 
systems. 

Given  the  shortfalls  of  the  current  heuristic  test  planning  approach  described  in 
the  previous  section,  the  objectives  of  this  thesis  are  to: 

•  Identify  a  formalized  process  model  that  accurately  describes  the  decision 
process  used  to  control  a  simplified  acoustic  range  event. 

•  Evaluate  the  utility  of  this  model  for  assisting  the  range  manager  in 
conducting  pre-event  decision  analysis  and  risk  management. 

This  thesis  poses  the  question:  can  formally  modeling  the  business  process  of 
conducting  test  range  events  expose  previously  overlooked  ambiguities  and  high-value 
decision  points?  To  support  conclusions  about  this  question,  the  thesis  will  compare  the 
number  of  unique  decision  points  and  the  total  number  of  events  in  each  model. 

C.  RESEARCH  METHODOLOGY 

The  basic  approach  of  the  thesis  research  methodology  is  to: 

•  Conduct  an  analysis  of  possible  decision  modeling  approaches  that  could 
assist  acoustic  range  test  planners  in  identifying  high-value  decisions  that 
occur  during  test  events. 

•  Identify  a  simplified  process  model  that  represents  the  current  acoustic 
range  test  event  decision  methodology. 

•  Create  a  formal  process  model  that  improves  on  the  simplified  decision 
model  through  the  use  of  a  formal  model  to  determine  if  high-value 
decision  nodes  can  be  identified  and  planned  for  in  advance  of  test  runs. 

•  Recommend  possible  uses  of  the  model  to  assist  range  event  planners. 
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D.  THESIS  ORGANIZATION 


Chapter  I  introduces  the  problem  faced  by  test  range  managers  and  posed  the 
research  question  the  thesis  researched. 

Chapter  II  reviews  business  process  modeling  literature  and  selects  a  suitable 
approach  for  documenting  the  current  decision  process  of  the  test  range  at  NUWC 
Division,  Keyport. 

Chapter  III  analyzes  current  NUWC  Division,  Keyport  range  decision-making 
processes,  proposes  a  simplified  model  to  represent  current  decision  approaches,  and 
identifies  weakness  in  the  model  requiring  improvement. 

Chapter  IV  develops  a  formalized  model  of  the  range  decision  process,  discusses 
model  runs  within  the  selected  software  tool,  and  provides  information  generated  from 
completed  model  runs. 

Chapter  V  analyzes  results  of  model  execution  and  provides  conclusions  on  the 
usefulness  of  the  model  to  range  managers.  The  chapter  concludes  with 
recommendations  for  model  improvement  and  further  study. 
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II.  REVIEW  OF  BUSINESS  PROCESS  MODELING 

LITERATURE 

This  chapter  reviews  decision-modeling  literature  with  the  purpose  of  identifying 
approaches  or  languages  that  would  assist  in  modeling  the  business  and  decision-making 
process  of  conducting  an  acoustic  range  test.  The  chapter  describes  a  sample  of  current 
modeling  approaches  and  provides  evaluation  of  potential  choices  to  represent  decision 
processes  used  during  acoustic  range  T&E  events.  Examination  of  other  process  models 
includes  review  of  potential  utility  for  assisting  thesis  research.  The  chapter  concludes 
with  the  selection  of  potential  models  or  processes  that  will  assist  in  modeling  a  simple 
acoustic  T&E  event. 

A.  BUSINESS  PROCESS  MODELS 

Business  process  models  (BPM)  or  process  modeling  is  a  type  of  workflow 
concept  that  separates  work  activities  into  defined  tasks,  roles,  and  procedures  that 
describe  that  organization’s  method  of  fulfilling  customer  needs  (Georgakopoulos, 
Homick,  and  Sheth  1995).  Process  models  range  in  complexity  from  simple  two- 
dimensional  workflow  charts  to  customizable  software  simulations. 

1.  Process  Model  Approaches 

Several  process  models  are  available  that  are  capable  of  representing  complex 
systems.  The  following  sections  describe  and  evaluate  potential  business  process  models 
(BPMs)  that  could  potentially  serve  as  a  process  for  range  event  decision-making. 

a.  Business  Process  Modeling  and  Notation 

Business  process  modeling  and  notation  (BPMN)  is  a  process  diagram  approach 
using  on  flowcharting  methods  to  create  graphical  representations  of  business  processes 
(White  2004;  Reisig  2013).  White  notes  that  use  of  graphical  elements  allows  for  simple 
diagrams  that  still  capture  complex  business  processes,  and  that  limiting  the  diagram 
elements  to  a  small  of  number  shapes  and  connectors  precludes  modelers  having  to  learn 
complex  notation.  BPMN  is  a  modeling  approach  that  if  applied  correctly  can  provide 
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complete,  efficient,  understandable,  and  sustainable  models  (Geyer  and  Fourie  2015). 
Modelers  have  used  BPMN  to  study  information  systems  (Wohed  et  al.  2006),  healthcare 
(Ojo  2012),  and  software  development  (Ballantine  2016).  Modelers  must  carefully  select 
model  grammar  to  avoid  user  rejection  of  the  model,  due  to  some  limitations  of  BPMN  to 
reflect  real  world  business  rules  (Recker  et  al.  2010).  This  constraint  may  limit  the 
usefulness  of  applying  BPMN  to  range  decision-making  due  to  its  heavy  use  of  real- 
world  heuristics  in  defining  decision  processes. 

b.  Petri  Nets 

Petri  Net  Markup  Language  (Hillah  et  al.  1970;  Desel  and  Reisig  2015)  provides 
implementation  for  the  Petri  net  process  flow  method.  Petri  nets  are  widely  employed  and 
have  their  own  International  Standard  that  ensures  common  usage  (Hillah  et  al.  2009). 
Mehrez  et  al.  (1995)  demonstrated  the  viable  use  of  Petri  Nets  for  modeling  behavioral 
processes  and  decision-making  and  showed  how  to  make  simple  decision  trees.  The 
graphical  visualization  of  the  Petri  net  might  be  useful  as  an  abstraction  of  the  range 
event  decision-making  process.  However,  according  to  the  NUWC  Division,  Keyport 
Range  Manager,  range  managers  and  test  planners  do  not  receive  training  in  modeling 
techniques  or  in  using  abstract  methods  to  plan  or  conduct  range  events.  These 
stakeholders,  therefore,  may  not  readily  accept  Petri  nets  modeling  as  a  basis  for 
evaluating  their  decisions. 

c.  Process  Algebra 

Process  algebras  such  as  Pi  and  Lambda  Calculus  can  model  concurrent 
communication  threads  (Wing  2002)  which  themselves  are  similar  to  behavior  processes. 
Baeten  and  Weijland  (1990)  showed  process  algebra  application  to  high-level  modeling. 
Wang  and  Ruhe  further  state  real-time  process  algebra  (2009,  137)  can  model  cognitive 
processes  such  as  behaviors  and  sequences  of  actions. 

Although  a  powerful  tool,  process  algebras  require  proficiency  in  building 
notation  and  some  understanding  of  programming.  The  goal  of  the  thesis  is  to  build  a 
process  model  that  range  managers  can  use  with  minimal  training.  They  generally  have 
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little  software  programming  experience;  therefore,  the  process  algebra  approach  may  be 
more  complex  than  the  problem  requires. 

d.  Yet  Another  Workflow  Language  (YAWL) 

The  YAWL  approach  uses  a  tree-like  hierarchical  structure  through  connecting 
high-level  Petri  nets  to  simulate  system  processes,  states,  and  transitions  (YAWL  2016). 
YAWL  can  represent  process  flows  similar  to  decision  trees  (van  der  Aalst  and  ter 
Hofstede  2005).  Although  YAWL  has  a  relatively  high  level  of  abstraction,  it  is  primarily 
graph  oriented  (Ouyang  et  al.  2006)  whereas  the  range  decision  process  is  more  akin  to  a 
block  diagram  approach.  An  abstract,  graphical  approach  may  not  be  suited  to  identifying 
high-value  decisions  in  a  format  acceptable  by  acoustic  range  operators  and  planners. 

e.  Monterey  Phoenix 

Monterey  Phoenix  is  a  BPM  approach  that  also  provides  high-level  abstraction 
within  a  behavior-modeling  framework  (Farah-Stapleton  and  Auguston  2013).  MP  can 
model  events,  activities  and  sequences.  Additionally,  it  is  not  constrained  by  the  single 
flowchart  template,  which  can  lead  to  over-constrained  process  models.  (Auguston  et  al. 
2015).  Use  of  Monterey  Phoenix  to  model  software  architecture  and  other  systems  has 
been  successful  (Farah-Stapleton,  Auguston,  and  Giammarco  2016).  Monterey  Phoenix  is 
well  suited  for  business  process,  system  and  software  modeling,  but  to  make  it  accessible 
to  untrained  range  personnel  would  require  more  tutorials  and  educational  materials  than 
were  available  at  the  time  of  this  writing. 

/.  Enhanced  Functional  Flow  Block  Diagrams 

Enhanced  Functional  Flow  Block  Diagrams  (EFFBD)  build  on  traditional  functional 
block  diagrams  by  adding  data  flow  overlays  to  provide  accurate  representation  of  data 
dependency  (Long  2002).  Long  notes  that  EFFBD  allows  modelers  to  use  either  control 
or  data  triggers  (or  both)  to  initiate  system  functions,  and  provides  status  of  trigger  data 
queuing.  EFFBDs  provide  descriptions  of  system  functions  and  their  execution  order,  and 
provide  many  constructs  useful  to  modelers  in  representing  processes  (Seidner  and  Roux 
2008).  EFFBD  tools  such  as  CORE  provide  software  that  aids  state  transition  mapping 
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and  behavioral  modeling  (Vitech  2016)  and  supports  the  construction  of  system  models 
that  can  perform  behavioral  analysis  (Rodano  and  Giammarco  2013).  CORE  supports 
multiple  users  and  allows  them  to  add  and  delete  to  the  model  through  a  centralized 
repository  (Alghazzawe  et  al.  2014). 

g.  Action  Diagrams 

Action  diagrams,  activity  diagrams  or  functional  flow  diagrams  are  methods  of 
displaying  action  entities  and  their  interactions  with  each  other  through  input/output, 
resources,  and  logic  flow  constructs  (SPEC  Innovations  2017).  These  approaches 
originated  with  the  development  of  the  Unified  Modeling  Language  (UML),  which 
provided  better  representation  of  the  development  of  software  systems  but  have 
application  to  other  business  processes  (Chen  et  al.  2007).  As  described  by  Chen  et  al., 
activity  and  action  diagrams  contain  activity  states  that  compute  outputs  based  on  their 
programming,  and  then  pass  their  result  on  to  the  next  activity  state  in  the  diagram.  Many 
UML  software  packages  provide  multiple  architecture  views,  models,  examples, 
allocation  and  consumption  of  resources,  and  contains  preloaded  formats  that  match 
DOD  Acquisition  Framework  views  (Pospisal  2015).  Action  diagrams  allow 
deconstruction/decomposition  of  more  complex  actions  to  better  illustrate  interactions 
between  activities,  functions,  and  tasks  (SPEC  2016). 

Action  diagram  software  such  as  SPEC  Innovation’s  Innoslate  tool  provide 
model-based  systems  engineering  tools  that  can  manage  requirements,  model  behaviors, 
and  represent  simple  to  complex  processes  (Rodano  and  Giammarco  2013).  Innoslate 
also  allows  discrete  event  simulation  (Ritter  and  Sperlazza  2016)  which  simplifies 
construction  of  a  formal  model  that  could  represent  the  decision  process  used  during  a 
range  test  event. 

2.  Selection  of  Process  Model  Approach 

All  of  the  modeling  approaches  reviewed  could  potentially  represent  range 

decision  processes.  Each  has  sufficient  grammar  constructs  to  capture  range  test  event 

details,  and  many  have  software  applications  available  that  allow  simulation  modeling. 

However,  a  primary  goal  of  the  thesis  is  to  develop  a  model  with  a  high  chance  of 
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acceptance  by  range  managers  and  operators  who  are  untrained  in  formal  modeling 
approaches  or  modeling  software  applications.  According  to  the  NUWC  Division, 
Keyport  Range  Manager,  these  personnel  use  graphical  approaches  such  as  cause 
mapping,  fault  tree  analysis,  and  block  diagrams  for  both  training  and  execution  of  range 
operations.  Therefore,  a  process  model  that  reflects  these  attributes  would  have  a  higher 
probability  of  adoption  by  range  personnel.  Modeling  approaches  that  require  high 
abstraction  or  knowledge  of  modeling  theory  would  likely  encounter  resistance  to 
adoption. 

Action  or  activity  diagrams  are  capable  of  representing  decision  processes  used 
by  range  managers  and  ROs,  and  their  usefulness  in  decomposing  processes  allows 
sufficient  detail  of  the  various  decision  activities  encountered  during  range  events.  The 
simplified  grammar  available  in  action  diagrams  may  also  be  useful  in  allowing  untrained 
range  operators  to  see  and  understand  the  functional  flows  of  the  model,  which  should 
increase  the  likelihood  of  the  use  of  the  model  in  assessing  test  event  risk.  Available 
software  simulation  applications  provide  sufficiently  simple  interfaces  that  range 
managers  could  use  in  their  planning  activities.  Therefore,  the  thesis  uses  an  action- 
diagram  based  approach  for  developing  the  formal  model  representing  range  event 
decision  processes 

B.  SELECTION  OF  SIMULATION  MODELING  TOOL 

This  thesis  uses  Innoslate  to  provide  a  formal  model  of  the  acoustic  range 
decision-making  process.  The  author  developed  previous  modeling  architecture  for  the 
acoustic  range  using  Innoslate,  thus  allowing  the  thesis  to  benefit  from  that  work. 
Innoslate  provides  flexibility  in  representing  behavioral  processes  and  has  a  relatively 
simple  user  interface.  Provided  tools  in  the  software  allow  for  straightforward 
representation  of  decision  points  and  other  range  behaviors,  and  previous  work  has 
shown  a  straightforward  formal  model  of  the  range  decision  process  is  possible  using  the 
action  diagram  notation. 

This  chapter  reviewed  possible  modeling  approaches  for  application  in  describing 
the  decision  process  of  an  acoustic  range  event.  The  review  focused  on  decision  models 
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that  could  represent  the  decision  process  of  the  acoustic  range  test  event.  Examination  of 
other  process  models  was  included  to  ensure  consideration  of  a  wide  range  of  approaches. 
The  review  also  highlighted  factors  that  would  contribute  to  adoption  of  a  formal  model. 
Since  range  managers  and  ROs  would  have  little  training  in  process  modeling  languages 
or  abstract  approaches,  they  would  be  more  likely  to  accept  activity  diagrams  with  simple 
modeling  grammar.  The  chapter  concluded  with  the  selection  of  the  action  diagram 
notation  and  modeling  approach  as  implemented  in  the  life  cycle -modeling  tool  Innoslate 
for  describing  the  acoustic  range  decision  process. 
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III.  ANALYSIS  OF  CURRENT  TEST  RANGE  DECISION 

PROCESSES 

The  purpose  of  this  chapter  is  to  provide  a  description  of  the  decision  process 
used  to  control  test  events  on  the  acoustic  range  and  to  provide  a  preliminary  model  to 
represent  current  knowledge  about  this  process.  The  simple  model  describes  a  basic 
underwater  test  and  uses  a  process  flow  chart  to  map  the  decision  nodes,  connections,  and 
feedback  loops  to  show  the  current  state  of  decision-making.  The  chapter  breaks  this 
model  down  in  an  analysis  of  each  step,  providing  observations  on  limitations  of  the 
current  model  and  postulates  possible  high-value  decision  points  based  on  the  functions 
described. 

A.  ACOUSTIC  RANGE  MODEL  CURRENT  STATE 

Levchuk  et  al.  described  heterarchical  decision  processes  as  those  that  are 
interdependent  and  may  have  a  superior  decision-maker  based  on  abilities  or  the  mission 
(2005).  The  acoustic  range  current  decision  process  resembles  a  heterarchy  due  to  the 
interdependence  of  the  decision  nodes  and  multiple  actors  in  the  process.  According  to 
the  NUWC  Division,  Keyport  Range  Manager,  acoustic  range  test  events  involve  three 
phases:  planning,  execution,  and  analysis.  He  also  provided  the  following  synopsis  of 
each  phase  of  operations. 

1.  Test  Event  Planning  Phase 

In  the  planning  phase,  program  personnel  provide  a  test  plan  to  identify  test 
objectives,  usually  in  the  form  of  required  data  such  as  acoustic  measurements,  recording 
of  three-dimensional  real-time  tracking  of  the  unit  under  test  (UUT),  or  other  signals. 
Test  directors  (TD)  use  these  test  plans  to  develop  a  run  plan,  which  provides  specific 
instructions  on  setting  up  test  equipment,  position  of  launch  and  recovery  craft,  UUT 
settings,  and  timing  of  launches. 

Both  the  RO  and  TD  work  together  in  the  planning  phase  to  develop  the  run  plan. 

The  TD  ensures  the  run  plan  will  achieve  program  objectives,  which  usually  involves  the 

collection  of  data.  Examples  of  data  collected  include  acoustic  information  such  as  UUT 
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radiated  noise,  ambient  noise  from  the  environment,  acoustic  signals  such  as  sonar,  and 
tracking  information.  The  TD  designs  the  test  sequence  to  maximize  data  collection  in  the 
time  available  for  UUT  run,  usually  limited  by  the  test  unit’s  fuel  or  power  source.  The 
TD  usually  produces  a  draft  run  plan  and  then  provides  the  draft  to  the  RO  for  review. 

The  role  of  the  RO  in  test  planning  is  to  ensure  the  test  will  progress  safely  while 
achieving  program  objectives. 

Risk  mitigation  decisions  in  the  planning  phase  consist  of  establishing  safety 
protocols  to  prevent  the  UUT  from  striking  the  firing  ship,  in-water  sensors,  or  the 
seabed.  TDs  establish  depth  control  and  shutdown  settings  so  that  the  UUT  will  return  to 
its  required  depth  band  through  software  commands.  If  the  UUT  continues  to  operate 
outside  the  set  depth  band,  shutdown  settings  will  cause  the  unit  to  shut  off  its  propulsion 
system  and  abort  the  run.  TDs  use  the  combination  of  launch  position,  launch  heading, 
UUT  speed,  and  the  timing  of  course  changes  to  develop  run  geometries  that  assist  them 
in  predicting  where  the  UUT  will  be  at  any  time  during  the  test  run.  TDs  use  these 
geometries  to  manage  safe  ranges  and  positions  of  the  launch  vessel  and  test  sensors  as 
well  as  to  prevent  the  UUT  from  hitting  the  seabed  or  shoreline.  Use  of  this  approach 
coupled  with  the  high  reliability  of  UUT  control  and  shutdown  safeties  has  resulted  in  no 
instances  of  a  UUT  striking  a  firing  vessel  in  50  years  of  operations. 

2.  Test  Execution  Phase 

Once  planning  is  complete,  TDs  provide  the  run  plan  to  the  range  officer  (RO) 
who  uses  the  plan  to  develop  various  documents  that  provide  specific  instructions  to 
technicians  and  range  vessel  operators.  The  documents  communicate  information  such  as 
which  UUTs  to  launch  and  when,  changes  in  setpoints  from  the  original  run  plan,  and  any 
special  instructions  required  by  the  RO  to  control  the  test.  The  RO  issues  these 
documents  usually  one  to  two  days  in  advance  of  the  test  run  date. 

On  the  day  of  the  test,  the  RO  coordinates  the  actions  of  range  vessels,  support 
technicians,  and  other  personnel  to  establish  the  proper  initial  conditions  for  testing.  The 
RO  uses  a  countdown  type  procedure  (referred  to  as  the  countdown)  to  control  the  steps 
necessary  to  prepare  the  UUT  for  launch.  The  countdown  has  intentional  stops  or  hold 
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points  built  in  such  that  the  RO  prohibits  subsequent  actions  until  all  steps  prior  to  the 
hold  point  are  completed.  Range  operations  personnel  receive  specific  training  and 
instruction  on  following  countdown  protocol  to  ensure  this  discipline.  This  approach 
provides  the  RO  complete  control  over  the  progression  of  the  test  and  ensures  the  UUT 
launch  occurs  only  after  all  test  parameters  meet  run  plan  specifications.  The  decision¬ 
making  exercised  by  the  RO  in  this  phase  consists  of  his  evaluation  of  test  preparations  at 
each  step  of  the  countdown,  and  his  assessment  of  readiness  to  proceed  to  the  next  step. 

Once  the  RO  decides  preparations  are  complete,  he  gives  the  direction  to  launch 
the  UUT.  The  RO  then  monitors  the  test  run,  observing  for  normal  indications  such  as 
UUT  following  the  predicted  geometry,  UUT  speed,  and  depth.  If  the  RO  observes  any 
abnormalities,  the  RO  must  choose  from  several  options  available.  These  options  consist 
of  allowing  the  test  to  continue  with  no  action,  directing  the  transmission  or  cessation  of 
acoustic  signals,  or  if  the  UUT  is  equipped  with  wire  guidance,  the  shutdown  of  the  UUT. 
These  decision  options  reflect  that  not  every  abnormality  requires  UUT  shutdown,  as  the 
result  may  simply  be  loss  of  one  data  parameter. 

During  test  runs,  the  RO  consults  with  the  TD  when  possible  to  determine  which 
option  best  meets  program  objectives.  An  RO’s  training  requires  them  to  take  immediate 
action  without  consultation  if  in  their  opinion  personnel,  range  vessels,  test  assets,  or  the 
UUT  are  in  immediate  danger  of  injury  or  damage.  Such  action  may  result  in  aborting  the 
entire  test  run. 

3.  Analysis  Phase 

After  UUT  run,  analysis  teams  receive  data  such  as  three-dimensional  tracking 
recordings,  acoustic  measurements,  and  other  signal  recordings.  The  teams  use  this 
information  to  reconstruct  the  test  run  and  determine  whether  the  UUT  performed  as 
expected.  If  the  teams  determine  UUT  test  run  problems,  they  provide  reports  to  program 
managers  on  possible  root  causes  and  recommended  actions.  These  recommendations 
often  result  in  adjustments  to  software,  hardware,  or  procedural  documents.  If  any 
problems  are  a  result  of  mistakes  made  by  range  operations  personnel,  the  teams  provide 
this  information  to  the  range  manager  for  corrective  action. 
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B.  TEST  EVENT  SIMPLIFIED  MODEL 


Figure  1  provides  a  simplified  flow  chart  model  of  the  decision  process  exercised 
by  the  range  officer  during  the  execution  phase  of  the  test  event.  The  simplified  model 
represents  the  basic  elements  of  the  current  decision-making  process  used  by  the  RO  in 
controlling  range  events.  As  previously  discussed,  the  model  represents  a  notional  event 
involving  the  launch  of  a  self-powered  high-speed  test  unit  with  a  run  time  of 
approximately  15  minutes. 


Figure  1 .  Range  Event  Flow  Chart 


1.  Model  Deconstruction 

The  scope  of  the  thesis  examines  the  decision  processes  used  from  the  time  of 
UUT  launch  until  the  end  of  the  test  run.  As  previously  discussed,  the  test  run  ends  either 
when  the  UUT  runs  out  of  fuel  or  receives  a  shutdown  command  ordered  by  the  RO.  The 
RO  uses  parameters  such  as  UUT  position  on  range,  course,  speed,  depth,  and  acoustic 
signals  generated  by  the  UUT  to  decide  if  action  is  required.  In  these  situations,  the  RO 
may  choose  to  take  no  action.  The  following  sections  describe  where  decision  points 
occur  in  the  model. 
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a. 


Begin  Event  /  UUT  Launch 


The  model  begins  with  the  launch  of  the  UUT  after  all  launch  preparations  of 
made.  The  RO  uses  the  countdown  procedure  to  ensure  all  launch  preparations  such  as 
UUT  readiness,  vessel  heading  and  speed,  and  position  of  acoustic  recorders  and  assets  in 
their  proper  locations.  Once  the  RO  verifies  these  parameters  are  per  the  run  plan,  he 
gives  the  order  to  launch  the  UUT.  The  decision  points  include  whether  the  UUT  meets 
launch  parameters,  and  whether  range  setup  (vessels,  weather)  supports  launch. 

b.  Monitor  Performance 

After  UUT  launch,  the  RO  monitors  indications  to  verify  the  UUT  has  started  and 
is  running  as  expected.  Three-dimensional  real-time  track  data  is  the  primary  indication 
used  by  the  RO  to  monitor  UUT  run  progress.  Acoustic  emanations  from  the  UUT  also 
provide  indications  of  particular  software  routines  in  progress  onboard  the  UUT, 
providing  further  information.  This  step  of  the  process  represents  information  gathering 
and  so  does  not  contain  decision  points. 

c.  Test  Objectives  Met 

During  this  portion  of  the  event,  the  RO  compares  UUT  performance  to  the 
expected  behavior  provided  in  the  run  plan.  Major  milestones  include  the  UUT  changing 
course  at  preset  coordinates,  or  changing  course  due  to  interaction  with  acoustic  sources 
that  stimulate  UUT  software  actions.  If  the  RO  observes  the  UUT  not  acting  per  run  plan 
expectations,  he  then  moves  to  the  next  block.  Decision  points  in  this  portion  include 
determining  if  UUT  performance,  testing  devices,  vessels,  or  other  factors  are  or  remain 
sufficient  for  achieving  test  objectives  and  ensuring  safety. 

d.  Intervention  Required 

Once  the  RO  observes  an  abnormality,  he  must  decide  whether  to  intervene  in  the 
test  or  allow  it  to  continue.  The  RO  weighs  the  type  of  abnormality  against  the 
consequence  of  allowing  it  to  continue.  For  example,  a  test  may  be  evaluating  the  ability 
of  the  UUT  to  react  to  a  specific  sonar  frequency.  If  the  UUT  fails  to  react,  intervention 
may  not  be  required  since  the  test  itself  is  evaluating  that  exact  interaction.  The  test  may 
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provide  multiple  opportunities  for  such  interaction,  therefore,  if  the  RO  took  action  he 
would  be  interfering  with  testing  objectives.  If  intervention  is  not  required,  the  feedback 
loop  shown  represents  the  RO  returning  to  monitoring  the  remainder  of  the  event.  This 
portion  represents  a  high-value  point  in  that  if  the  RO  decides  that  UUT  or  range  safety  is 
about  to  be  compromised,  he  must  take  action  that  will  likely  end  the  test  run  prior  to 
completion  of  all  testing  objectives. 

e.  Take  Available  Action 

Once  the  RO  decides  action  is  required,  he  then  must  decide  what  action  to  take. 
Decision  options  include  no  action,  adjustment  of  acoustic  sources  to  stimulate  UUT 
software  actions,  or  directing  a  shutdown  command  (if  the  UUT  is  so  equipped)  that  will 
abort  the  test.  As  test  events  are  expensive,  ROs  generally  do  not  shutdown  UUTs  unless 
in  their  opinion  such  action  is  necessary  to  protect  personnel,  range  equipment  or  the 
UUT.  For  this  reason,  this  block  represents  a  critical  decision  point. 

Once  the  RO  takes  action,  he  then  evaluates  if  such  action  corrected  the  problem. 
The  feedback  loop  to  the  Test  Objectives  Met  block  represents  the  RO’s  subsequent 
analysis  of  the  effectiveness  of  his  corrective  action(s)  and  the  decision  process  again 
enters  the  “Intervention  Required”  decision  block. 

/.  Test  Objectives  Met:  Yes 

Once  the  RO  observes  that  the  event  has  achieved  as  many  test  objectives  as 
possible,  he  can  either  allow  the  UUT  to  run  to  fuel  exhaustion  or  command  its 
shutdown.  Depending  on  the  type  of  test,  a  test  objective  may  include  UUT  fuel 
exhaustion  because  as  long  as  the  UUT  has  fuel  it  continues  to  run.  This  requires  the  RO 
to  continue  monitoring  to  ensure  range  safety.  The  decision  point  in  this  portion  centers 
on  the  RO  determining  if  previous  action  was  sufficient  such  that  range  safety  and 
meeting  test  objectives  is  achievable. 

g.  Test  Event  Ends  /  UUT  Shutdown 

The  event  concludes  when  the  RO  determines  that  the  test  achieved  all  objectives 
or  the  UUT  shuts  down  from  fuel  exhaustion  or  command  shutdown.  The  RO  has  the 
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option  to  allow  the  UUT  to  continue  operations  even  if  the  test  accomplished  all 
requirements.  According  to  the  NUWC  Division,  Keyport  Range  Manager,  ROs  routinely 
allow  the  UUT  to  continue  running  even  if  equipped  with  command  shutdown,  as  this 
allows  program  managers  to  have  more  data  that  may  benefit  test  analysis. 

2.  Summary  of  Decision  Points  from  the  Simplified  Model 

•  Intervention  required.  The  RO  determined  overt  action  is  required  to 
protect  the  UUT  from  damage,  ensure  range  safety,  or  prevent 
unnecessary  loss  of  test  data  due  to  a  change  in  test  conditions  or  UUT 
performance 

•  Take  appropriate  action.  The  RO  decides  what  action  to  take  in  order  to 
protect  the  UUT  or  range  assets  or  avoid  loss  of  test  data 

•  Test  objectives  met.  The  RO  determines  that  all  test  objectives  have  been 
met  and  the  test  may  conclude,  either  by  the  UUT  running  out  of  fuel  or 
through  a  command  shutdown  of  the  UUT  if  so  configured 

3.  Limitations  of  the  Simplified  Model 

The  simplified  model  is  accurate  but  lacks  the  detail  necessary  to  represent  the 
decision  process  during  a  test  event.  The  following  paragraphs  provide  the  improvements 
required  in  order  to  model  the  process  adequately. 

a.  Lack  of  Representation  of  Launch  Preparation  Decisions 

The  simplified  model  decision  process  begins  with  the  decision  to  launch  based 
on  completion  of  the  countdown  procedure.  According  to  the  NUWC  Division,  Keyport 
Range  Manager,  the  RO  must  make  other  decisions  prior  to  this  point  depending  on  UUT 
start  up  performance,  range  conditions  such  as  vessel  performance,  launch  system 
readiness,  and  weather.  Many  of  these  decisions  can  affect  the  outcome  of  the  test  event 
and  therefore  require  representation  in  the  model. 


19 


b.  Influence  of  Range  Environmental  Conditions  on  Continuance  of 
Testing 

Weather,  sea  state,  boating  traffic,  ambient  noise,  and  marine  mammal  activity 
also  influence  the  RO’s  decision  to  authorize  UUT  launch.  The  model  lacks  the 
granularity  necessary  to  represent  these  decision  factors. 

c.  Differences  between  Decision  Types 

The  model  does  not  distinguish  between  the  decision  to  allow  the  test  to  continue 
after  intervention  (i.e.,  while  test  objectives  remain)  and  the  decision  to  end  the  event 
after  accomplishment  of  required  test  objectives.  These  decisions  are  of  significant 
difference  to  warrant  separate  representation. 

d.  Possible  Outcomes  of  Taking  Action 

The  model  assumes  that  if  the  RO  takes  action,  the  resulting  sub-process  is  the 
determination  whether  the  event  is  meeting  test  objectives.  However,  the  RO  may  need  to 
take  action  that  will  immediately  end  the  event  due  to  safety  or  other  concerns.  The 
model  does  not  reflect  this  possible  decision  outcome. 

This  chapter  analyzed  a  preliminary  process  flow  chart  that  described  current 
knowledge  about  a  basic  underwater  test  event  at  the  acoustic  range.  The  chapter 
identified  existing  critical  decision  points  represented  in  the  model  and  the  purpose  of 
these  decisions.  The  chapter  concluded  with  an  analysis  of  limitations  of  the  simplified 
model  in  order  to  influence  a  more  accurate  representation  in  the  subsequent  formal 
model. 

4.  Simplified  Model  Validation 

Review  of  the  simplified  model  with  the  NUWC  Division,  Keyport  Range 
Manager  confirmed  it  represented  the  basic  elements  of  the  range  decision  process. 
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IV.  CONSTRUCTION  OF  A  FORMALIZED  TEST  RANGE 

MODEL 


This  chapter  provides  a  formal  model  of  the  preliminary  process  model  presented 
in  Chapter  II  using  the  Innoslate  software  application.  This  formal  model  captures  the 
Chapter  II  analysis  recommendations  for  improving  the  precision  and  completeness  of 
the  decision  flow  process  for  the  subject  acoustic  range  test  event.  The  formal  model 
describes  and  includes  additional  decision  nodes,  which  expands  the  set  of  possible 
decision  flows  for  consideration  in  advance  of  an  actual  test  event.  The  chapter  also 
provides  a  discussion  of  the  modeling  framework  and  construction  of  the  model  as  well 
as  results  of  several  modeling  runs.  The  chapter  concludes  with  observations  on  the 
model  run  results  and  possible  implications  for  actual  acoustic  test  range  events. 

The  model  uses  generic  language  to  describe  decisions,  choices,  and 
consequences  of  the  choices.  For  example,  a  decision  point  preceded  by  “take  available 
action,”  followed  by  the  decision  point  labeled  “action  effective?”  followed  by  outcomes 
of  yes  or  no.  This  allows  the  model  to  use  a  simplified  construct  for  evaluating  the 
decision  process  that  also  is  generic  enough  to  represent  any  yes/no  decision  type. 
Labeling  the  consequences  of  each  outcome  so  that  the  user  understands  the  implications 
of  each  decision  choice  allows  users  to  understand  these  simple  decisions  in  their  own 
specific  vernacular.  This  approach  allowed  for  a  reasonable  expedient  construction  of 
model  flow  regardless  of  the  type  of  mission  supported.  Limiting  the  initial  modeling  to 
yes/no  decisions  also  aided  in  identifying  clear  consequences  of  actions  so  that  users  had 
less  trouble  in  visualizing  the  decision  flow. 

A.  IMPROVEMENTS  TO  THE  SIMPLIFIED  MODEL 

The  simplified  model  has  several  weaknesses  that  require  improvement  to 
describe  accurately  the  test-event  decision  process.  Table  1  below  outlines  modifications 
required.  Locations  correspond  to  the  points  shown  in  Figure  2. 
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Table  1.  Simplified  Model  Improvements 


Modification 

Reasoning 

Figure  2 
Position 

UUT  preparation  supports 
launch 

Reflects  verification  of  necessary  UUT 
initial  conditions 

1 

Range  setup 

Data  recorders,  vessels,  weather,  ambient 
noise  conditions  must  all  meet  set 
parameters  prior  to  launch 

2 

UUT  performance  as 
expected 

Current  decision  point  (Test  Objectives 
Met)  focuses  on  test  objectives  only,  and 
does  not  reflect  other  factors  such  as 
UUT  malfunction  which  may  require  RO 
decision/action 

3 

Test  run  completion 
decision  block 

Simplified  model  does  not  distinguish 
between  test  objectives  occurring  during 
the  event  (i.e.,  test  event  milestones)  and 
completion  of  the  test.  A  separate 
feedback  loop  is  needed  to  continue  the 
test  if  objectives  are  not  met  versus 
deciding  if  all  objectives  are  met. 

4 

Test  abort 

The  RO  requires  the  ability  to  abort  the 
test  at  any  time  due  to  UUT  problems  or 
range  safety  issues 

5 
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Figure  2.  Simplified  Model  Corrections 


B.  MODEL  ASSUMPTIONS 

The  formalized  model  assumes  the  following  initial  conditions  and  test 
parameters  exist  prior  to  simulation  runs. 

•  The  RO  has  access  to  all  information  necessary  to  make  decisions. 
Situations  where  needed  information  is  unavailable  or  in  error  are  not 
modeled  in  this  version. 

•  The  RO  has  the  requisite  training  to  recognize  each  decision  point. 

•  The  RO  is  aware  of  possible  decision  choices  at  each  point,  including  the 
no-action  alternative. 

•  The  RO  is  aware  of  only  the  information  available  at  the  time  of  the 
decision,  i.e.,  the  model  does  not  allow  for  hindsight. 

•  The  RO  makes  decisions  at  each  point  only  once  and  does  not  alter  these 
decisions  unless  allowed  by  the  model  construct. 

•  UUT  performance  follows  historical  experience,  i.e.,  the  model  does  not 
introduce  previously  unknown  behaviors. 
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C.  FORMAL  MODEL  FRAMEWORK 

The  thesis  establishes  the  formal  model  framework  by  beginning  with  the 
simplified  model  and  adding  improvements  necessary  to  capture  the  major  decision 
elements  in  a  typical  acoustic  test  range  event.  The  formal  model  uses  the  Innoslate 
software  tool  to  replicate  decision  points,  necessary  resources  for  those  decisions, 
feedback  loops,  and  inputs/outputs.  The  following  life  cycle  modeling  language  (LML) 
constructs  available  in  Innoslate  are  used: 

•  Action  blocks  represent  the  various  functions  in  the  model.  They  allow 
the  model  to  set  initial  conditions  and  outcomes  for  decisions  blocks  (OR 
functions).  They  can  also  be  set  to  provide  criteria  for  entering  and  exiting 
LOOP  functions  through  use  of  script  in  the  Innoslate  tool.  Action  blocks 
provide  a  method  for  describing  the  consequences  of  outcomes  from 
decision  blocks,  and  are  therefore  useful  in  assisting  users  of  the  model  to 
understand  the  result  of  decision  choices. 

•  LOOP  functions  are  action  blocks  that  provide  a  recursive  activity  that 
repeats  a  series  of  imbedded  actions  until  conditions  meet  an  exit  criterion. 
Script  in  Innoslate  can  be  set  to  monitor  each  action  block  within  the  loop 
for  various  criteria,  which  can  be  set  to  either  continue  or  exit  the  loop. 
The  formal  model  construct  uses  basic  true/false  values  in  these  action 
blocks  to  determine  if  the  loop  will  end  or  continue  running,  depending  on 
the  outcome  of  imbedded  OR  decision  points. 

•  OR  blocks  are  another  type  of  action  block  that  provide  decision 
functions  within  the  formal  model.  Script  can  be  set  so  that  either  a 
probability  function,  the  presence  of  an  input  (resource),  or  a  customized 
set  of  conditions  determine  the  decision  outcome.  To  avoid  an 
unnecessarily  complex  model,  OR  outcomes  in  the  formal  model  were 
limited  to  simple  yes/no  criteria  with  probability  values  assigned  to  each 
outcome.  During  simulation  runs,  Innoslate  uses  a  random  number 
generator  function  to  determine  a  value  within  a  normal  probability 
distribution  and  then  compares  that  value  to  the  yes/no  probability  values 
set  for  that  OR  block. 

D.  MODEL  CONSTRUCTION 

1.  Replication  of  the  Simplified  Model 

Figure  3  shows  the  simplified  model  constructed  in  the  Innoslate  tool.  As 
described  in  section  B  of  this  chapter,  Figure  3  differs  from  Figure  1  due  to  the  simplified 


24 


model  decision  point  of  Test  Objectives  Met  did  not  distinguish  between  individual  test 
objective  issues  and  overall  test  objectives. 


Figure  3.  Simplified  Model  in  Innoslate 

To  allow  the  model  to  function  in  simulation,  the  Monitor  UUT  Performance 
block  contains  the  exit  criteria  for  the  overall  test  (test  completion).  As  discussed 
previously,  Innoslate  provides  simple  programming  language  to  assign  time  durations, 
probabilities,  loop  exit  conditions,  and  test  abort  criteria.  These  features  allow  for 
necessary  modifications  to  the  simplified  model  to  provide  a  closer  representation  of  the 
actual  decision  process  used  by  the  RO  during  an  acoustic  test  event. 
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2.  Model  Improvements 

Figures  4  and  5  demonstrate  the  improvements  made  to  the  simplified  model  so 
that  it  more  closely  represents  the  current  decision  process  used  for  acoustic  events. 
Figure  4  shows  the  addition  of  a  node  to  verify  initial  conditions.  Per  Table  1,  the  RO 
must  evaluate  several  environmental  and  technical  conditions  prior  to  allowing  launch. 
For  example,  weather  and  sea  states  must  support  not  only  the  launch  of  the  UUT  but 
also  the  ability  to  retrieve  the  UUT  once  its  powered  run  is  complete.  Technical 
conditions  such  as  UUT  warm  up,  entry  of  launch  system  parameters,  and  startup  of  data 
recording  systems  must  be  established  and  verified.  The  decision  to  proceed  with  UUT 
launch  has  high  value  because  failure  to  set  proper  conditions  can  result  in  an  aborted  test 
or  loss  of  valuable  data. 
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Figure  4.  Addition  of  Initial  Conditions  Decision  Block 

Figure  5  expands  and  modifies  the  Initial  Conditions  Met  section  to  represent 
more  accurately  the  actual  decision  made  by  the  RO  at  this  point  in  the  test.  The 
simplified  model  did  not  adequately  illustrate  the  recursive  decisions  the  RO  encounters 
during  range  and  UUT  preparations.  The  decision  node  as  modeled  provides  for  a  more 
accurate  simulation  of  the  duration  of  this  portion  of  the  test. 
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Figure  5.  Consolidation  of  Initial  Conditions  Decision 


Figure  6  shows  the  addition  of  a  decision  point  to  Figure  4  that  requires  the  RO  to 
decide  if  the  UUT  is  performing  per  expectations  (oval).  If  during  the  test  run  the  UUT 
malfunctions  or  behaves  in  an  unexpected  way,  the  RO  may  be  required  to  take  actions 
ranging  from  altering  the  UUT  run  to  aborting  the  test.  Inputs  to  this  decision  include 
environmental  factors  such  as  weather  and  ambient  noise,  as  well  as  UUT  performance 
parameters  such  as  speed,  depth,  course,  and  course  change  sequences.  This  addition 
address  the  weakness  identified  as  point  3  in  Figure  2. 


Figure  6.  Unit  under  Test  Performance 
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Figure  7  shows  further  modification  of  the  Monitor  UUT  Performance  section 
that  now  contains  the  criteria  for  exiting  the  Monitor  Performance  loop.  The  added 
language  (arrow)  defines  that  the  loop  continues  until  all  test  sequences  objectives  are 
accomplished.  The  following  OR  block  is  also  modified  to  reflect  monitoring  of  test 
sequence  objectives,  to  avoid  confusion  with  monitoring  for  overall  test  completion  (now 
accomplished  in  the  preceding  block).  This  modification  addresses  the  problem  identified 
as  point  4  in  Figure  2. 


^2^ — > 


Begin  Test 
Event 
(Launch) 


Figure  7.  Distinguishing  between  Test  Sequencing  and  Test  Completion 

Following  the  Test  Sequence  Objectives  Met  OR  block,  the  adding  of  additional 
branches  distinguishes  between  UUT  problems  and  test  asset/environmental  issues  that 
could  interfere  with  proper  test  sequencing  (see  oval).  Decomposition  of  each  branch 
appears  below. 

The  addition  of  Block  ID  numbers  to  each  block  aids  in  referencing  during 
construction  and  analysis.  The  Continue  Monitoring  Block  now  shows  a  number  in 
parenthesis  to  distinguish  this  block  from  other  blocks  in  the  model  with  the  same 
function.  The  model  repeats  this  convention  for  other  similar  block  labels.  Additionally, 
Innoslate  assigns  each  block  its  own  unique  ID  number,  which  assists  in  tracking  how  the 
model  executes  the  various  branches. 

Figure  8  shows  the  decomposition  of  the  Evaluate  UUT  Performance  branch.  OR 

nodes  represent  the  decision  points  encountered  by  the  RO  if  UUT  issues  arise  during  the 
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test,  and  the  subsequent  actions  for  these  decisions.  Scripts  in  the  OR  and  Action  blocks 
allow  the  model  to  progress  to  the  next  state  depending  on  decision  outcomes. 
Descriptions  of  the  decision  options  added  to  the  branches  describe  the  decision  options 
open  to  the  RO  at  each  node.  Action  blocks  now  contain  unique  identifiers  for  ease  of 
cros  s-referencing . 


Yes,  despite  the 


can  continue  safely 


Figure  8.  Evaluate  UUT  Performance  Branch 

Figure  9  shows  the  decomposition  of  the  Problem  with  Environment  or  Test 
Support  Assets  branch.  Similar  to  Figure  8,  test  abort  functions  or  the  continuance  of 
monitoring  are  now  provided  as  decision  outcomes.  Branch  description  language 
identifies  the  options  open  to  the  RO. 
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Figure  9.  Evaluation  of  Test  Asset/Environment 

The  additional  language  and  branches  in  Figures  8  and  9  now  provide  the  test 
abort  capability  required  by  the  RO,  and  address  the  problem  noted  in  point  5  of  Figure  2. 

Figure  10  shows  a  consolidation  of  the  two  main  sections  of  the  formal  model.  Adding 
this  layer  allows  for  the  use  of  the  decomposition  function  offered  in  Innoslate,  and  enables 
more  intuitive  use  and  exploration  of  the  model  by  potential  users.  The  Verify  Initial 
Conditions  block  decomposes  to  the  actions  described  in  Figure  5;  Conduct  Test  decomposes 
to  the  actions  in  Figure  7  with  subsequent  decompositions  shown  in  Figures  8  and  9. 


Figure  10.  Model  Consolidation 
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E.  MODELING  RUNS 


Once  the  model  was  constructed  and  debugged,  parameters  were  chosen  for  the 
various  probabilities  and  durations  found  in  each  of  the  decision  nodes  and  action  blocks. 
These  parameters  were  determined  through  interviews  and  walk-throughs  with  ROs  who 
conduct  test  events  on  the  acoustic  range  in  order  to  obtain  a  reasonable  baseline  of 
values.  For  ease  of  reference,  Appendix  A  to  the  thesis  contains  block  ID  designations, 
Innoslate  simulation  block  identifiers,  block  labels,  and  descriptions  of  the  functions  of 
each  block.  The  appendix  also  contains  the  initial  probability  and  duration  settings  for 
each  action,  LOOP,  and  OR  block  in  the  formal  model. 

1.  Model  Run  Setup 

Innoslate  provides  a  Monte  Carlo  simulation  function  that  allows  the  formal 
model  to  execute  multiple  runs  to  provide  statistical  results  of  the  decision  flows.  The 
model  used  one  thousand  iterations  per  run  to  achieve  a  reasonable  level  of  fidelity  in  the 
results  and  to  observe  trends.  Innoslate  provides  options  for  determining  the  outcome  of 
OR  decision  blocks  by  setting  discrete  probabilities  for  the  yes/no  choices,  or  to  select  a 
probability  function  such  as  normal,  uniform,  or  other  common  distributions.  For  ease  of 
model  set  up  and  straightforward  comparison  of  decision  block  usage,  the  model  used 
discrete  values  recorded  in  Appendix  A. 

2.  Model  Baseline  Runs 

Using  the  baseline  setpoints  for  the  model  found  in  Appendix  A,  the  model 
executed  10,000  simulation  runs  in  1,000-run  increments.  Innoslate  generated 
spreadsheets  for  each  run  that  provided  data  on  the  frequency  each  action  block  was 
active  during  an  individual  run.  Table  2  provides  the  summary  of  baseline  run  data  for  all 
blocks  in  the  formal  model. 
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Table  2.  Frequency  of  Action  Block  Usage  during  Baseline  Model  Runs 


Block  ID 

Action  Block  Label 

Frequency 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Average 

1 

Verify  Initial  Conditions 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

2 

Required  Initial  Conditions  Met? 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

3 

Proceed  to  UUT  Launch 

105 

100 

111 

98 

111 

108 

92 

94 

108 

91 

101.8 

4 

Establish  Proper  Conditions 

895 

900 

889 

902 

889 

892 

908 

906 

892 

909 

898.2 

5 

Conduct  Test 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

6 

Begin  Test  Event  (Launch) 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

7 

MonitorTest  Performance  Until  Test  Objectives  Complete 

1963 

1848 

1859 

1910 

1895 

1875 

1930 

1887 

1930 

1906 

1900.3 

8 

Test  Sequence  Objectives  Met? 

963 

848 

859 

910 

895 

875 

930 

887 

930 

906 

900.3 

9 

Continue  Monitoring  (1) 

759 

669 

693 

728 

714 

702 

752 

702 

742 

715 

717.6 

10 

UUT  Performance  or  Environmental  Test  Asset  Problem? 

204 

179 

166 

182 

181 

173 

178 

185 

188 

191 

182.7 

11 

Evaluate  UUT  Performance 

83 

76 

57 

68 

81 

64 

57 

66 

69 

87 

70.8 

12 

Evaluate  Env.  &Test  Support  Asset  Performance 

121 

103 

109 

114 

100 

109 

121 

119 

119 

104 

111.9 

13 

UUT  Performance  as  Expected? 

83 

76 

57 

68 

81 

64 

57 

66 

69 

87 

70.8 

14 

Abort  Required?  (1) 

73 

68 

52 

60 

71 

56 

54 

60 

63 

78 

63.5 

15 

Abort  Test  (1) 

7 

3 

5 

3 

4 

5 

5 

6 

2 

4 

4.4 

16 

Continue  Monitoring  (2) 

66 

65 

47 

57 

67 

51 

49 

54 

61 

74 

59.1 

17 

Take  Available  Action  (1) 

0 

8 

5 

8 

10 

8 

3 

6 

6 

9 

6.3 

18 

Action  Effective?  (1) 

0 

8 

5 

8 

10 

8 

3 

6 

6 

9 

6.3 

19 

Abort  Required?  (4) 

10 

10 

10 

12 

10 

3 

17 

16 

14 

8 

11 

19 

Continue  Monitoring  (3) 

1 

1 

1 

1 

2 

0 

1 

2 

0 

3 

1.2 

20 

Abort  Required?  (2) 

9 

7 

4 

7 

8 

8 

2 

4 

6 

6 

6.1 

21 

Abort  Test  (2) 

1 

2 

2 

0 

2 

1 

1 

0 

0 

1 

1 

22 

Allow  Test  to  Continue 

8 

5 

2 

7 

6 

7 

1 

4 

6 

5 

5.1 

23 

Problem  with  Environment  or  Test  Support  Assets? 

121 

103 

109 

114 

100 

109 

121 

119 

119 

104 

111.9 

24 

Take  Available  Action  (2) 

0 

93 

99 

102 

90 

106 

104 

103 

105 

96 

89.8 

25 

Action  Effective?  (2) 

0 

93 

99 

102 

90 

106 

104 

103 

105 

96 

89.8 

26 

Continue  Monitoring  (4) 

65 

59 

65 

68 

48 

65 

56 

59 

61 

55 

60.1 

27 

Abort  Required?  (3) 

46 

34 

34 

34 

42 

41 

48 

44 

44 

41 

40.8 

28 

Abort  Test  (3) 

5 

2 

4 

0 

3 

5 

4 

5 

4 

6 

3.8 

29 

Continue  Monitoring  (5) 

41 

32 

30 

34 

39 

36 

44 

39 

40 

35 

37 

31 

Abort  Test  (4) 

1 

1 

2 

1 

1 

0 

1 

2 

2 

0 

1.1 

32 

Continue  Monitoring  (6) 

9 

9 

8 

11 

9 

3 

16 

14 

12 

8 

9.9 

Table  2  shows  that  not  all  simulation  runs  resulted  in  the  use  of  all  action  blocks. 
Performance  of  multiple  sets  of  runs  revealed  that  all  decision  and  action  blocks  are 
active  at  various  times.  This  showed  that  all  model  paths  and  branches  are  active  and  are 
possible  outcomes.  Table  3  provides  information  on  the  frequency  of  appearance  of 
decision  blocks  only. 
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Table  3.  Action  Block  Frequency  during  Baseline  Runs 


FREQUENCY  OF  DECISION  BLOCK  APPEARANCE  DURING  BASELINE  RUNS 

Block 

ID 

Innoslate 

ID 

Block  Label 

Previous  Block 

Activity 

Function 

Description 

Frequency  of 
Appearance 

2 

e20PAl 

Required  Initial  Conditions 
Met? 

1 

OR 

Decision  node 

RO  verification  that  initial  conditions 
support  UUT  launch 

1000 

8 

e3WGC2 

Test  Sequence  Objectives 
Met? 

7 

OR 

Decision  node 

RO  verification  that  the  test  is 
progressing  per  the  run  plan 

900.3 

10 

e3XQEl 

UUT  Performance  or 

Environmental/Test  Asset 

Problem? 

8 

OR 

Action  if  outcome  to  8  is  NO 

RO  decision  on  whether  problem  is 
UUT  or  Environment/Test  Asset 

related 

182.7 

13 

e3WT4H 

UUT  Performance  as 
Expected? 

1 1 

OR 

Decision  node 

RO  decision  on  whether  UUT 
performance  is  as  expected  based  on 
the  run  plan  and  RO  experience 

70.8 

14 

e5T5 1 1 

Abort  Required?  (1) 

13 

OR 

Action  if  outcome  of  1 3  is  YES 

If  the  RO  observes  a  problem  but  the 
UUT  is  perfonning  normally,  he  must 
decide  if  a  test  abort  is  required 
regardless 

63.5 

18 

e6RBTH 

Action  Effective?  (1) 

17 

OR 

Decision  node 

RO  evaluation  of  corrective  action 

effectiveness 

6.3 

20 

e5SHG2 

Abort  Required?  (2) 

18 

OR 

Action  if  outcome  of  18  is  NO 

If  action  taken  is  ineffective,  RO  must 
decide  if  test  should  continue 

1.2 

23 

e5SHGl 

Problem  with  Environment 
or  Test  Support  Assets? 

12 

OR 

Decision  node 

RO  decides  if  test  assets  or 
environment  (e.g.  weather,  sea  state) 
are  a  problem 

5.1 

25 

e2QWH 

Action  Effective?  (2) 

24 

OR 

Decision  node 

RO  evaluation  of  corrective  action 

effectiveness 

89.8 

27 

e8KJBJ 

Aboil  Required?  (3) 

25 

OR 

Action  if  outcome  of  25  is  NO 

If  action  taken  is  ineffective,  RO  must 

decide  if  test  should  continue 

60.1 

30 

e21KKH 

Abort  Required?  (4) 

23 

OR 

Action  if  outcome  of  23  is  NO 

If  action  taken  is  ineffective,  RO  must 

decide  if  test  should  continue 

37 

3.  Sensitivity  Analysis 

The  thesis  research  question  centers  on  the  ability  of  a  formal  model  to  identify 
high-value  decision  points  in  a  decision  process.  To  assist  in  determining  if  the  formal 
model  can  provide  this  insight,  Appendix  B  documents  probability  adjustments  made  to 
two  decision  blocks  within  the  “Problem  with  Environment  or  Test  Support  Assets?” 
branch.  Block  25,  “Action  Effective  (2)”  yes/no  values  were  changed  from  60/40  to 
40/60,  and  Block  27  “Abort  Required  (5)”  yes/no  values  changed  from  10/90  to  30/70. 
Alterations  were  limited  to  this  branch  only  to  simplify  observing  changes  in  model 
execution  as  decision  parameters  were  changed.  The  model  executed  10,000  runs  with 
results  provided  in  Table  4. 

The  purpose  of  these  changes  was  to  verify  model  design  would  respond  as 
expected  given  simple  changes  to  the  decision  parameters.  Further  model  refinement 
would  include  quantitative  analysis  of  downstream  effects  of  changes  in  decision 
durations  and  probability  values. 
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Table  4.  Quantitative  Analysis  Showing  Sensitivity  of  Decision  Frequency  to 

Decision  Probabilities 


Block 

ID 

Block  Label 

Freq  ue  ncy  Ave  rages 
After  Changes 

Baseline  Average 
(From  Appendix  B) 

%  Change 

2 

Required  Initial  Conditions  Met? 

1000 

1000 

0.0% 

8 

Test  Sequence  Objectives  Met? 

895.3 

900.3 

0.6% 

10 

UUT  Performance  or  Environmental  Test  Asset  Problem? 

179.3 

182.7 

1.9% 

13 

UUT  Performance  as  Expected? 

72.5 

70.8 

2.4% 

14 

Abort  Required?  (1) 

64.7 

63.5 

1.9% 

18 

Action  Effective?  (1) 

7.8 

6.3 

23.8% 

20 

Abort  Required?  (2) 

5.5 

1.2 

358.3% 

23 

Problem  with  Environment  orTest  Support  Assets? 

106.8 

5.1 

1994.1% 

25 

Action  Effective?  (2) 

97.2 

89.8 

8.2% 

27 

Abort  Required?  (3) 

59.5 

60.1 

1.0% 

30 

Abort  Required?  (4) 

9.6 

37 

74.1% 

Table  4  shows  the  results  of  making  the  changes  indicated  in  Appendix  B. 
Probability  values  of  OR  blocks  25  and  27  were  the  only  changes  made  to  the  model.  As 
Table  4  indicates,  the  frequency  values  for  these  blocks  changed  little,  but  other  decision 
blocks  experienced  large  changes  in  their  frequency  of  appearance,  especially  18,  20,  23 
and  30. 

This  chapter  introduced  a  formalized  model  used  to  represent  a  simplified  test 
event  on  the  acoustic  range.  Discussion  of  assumptions,  framework,  parameters,  and 
construction  explained  how  the  model  represents  a  range  event  in  terms  of  the  decision 
flow  used  to  control  the  test.  Description  of  the  stages  of  model  construction  displayed 
modifications  of  the  simplified  model  that  result  in  formulation  of  a  formal  model  using 
the  Innoslate  simulation  tool.  Results  of  modeling  runs  provided  data  useful  in 
determining  the  model’s  effectiveness  at  representing  process  flows  and  decision  nodes 
during  a  test  run.  The  chapter  concluded  with  a  limited  sensitivity  analysis  that  showed 
how  adjustments  to  model  decision  probabilities  resulted  in  changes  to  the  frequency  in 
which  the  model  used  specific  decision  points. 
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4.  Model  Validation 


As  with  the  simplified  model,  the  NUWC  Division,  Keyport  range  management 
reviewed  the  formalized  model  and  confirmed  it  adequately  represented  a  routine  test 
event.  They  provided  several  improvement  recommendations  for  further  study  that  would 
assist  the  range  in  better  test  planning.  These  recommendations  are  included  in  the 
following  chapter. 


35 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


36 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 
FOR  FURTHER  RESEARCH 

The  purpose  of  this  chapter  is  to  provide  conclusions  regarding  the  research 
question  postulated  in  this  thesis,  in  light  of  the  analyses  presented  earlier.  The  chapter 
provides  findings,  resolves  the  thesis  question,  and  offers  recommendations  for  model 
use  and  further  research  that  may  assist  NUWC  Keyport  range  personnel  and  those 
working  in  other  T&E  domains.  The  chapter  concludes  with  recommendations  for  further 
study. 

A.  FINDINGS 

Findings  focus  on  the  ability  of  the  formal  model  to  represent  a  simple  range 
event,  the  ability  of  the  model  to  predict  high-value  decision  points,  and  the  usefulness  of 
the  formal  model  as  potential  tool  to  range  managers. 

1.  Modeling  of  a  Routine  Range  Event 

Improvements  to  the  simplified  model  resulted  in  a  formal  model  that  better 
described  the  actual  decision  process  used  during  test  events.  This  also  allowed  easier 
replication  of  the  process  when  building  the  model  in  Innoslate.  The  following 
paragraphs  describe  specific  comparisons  of  the  two  models. 

a.  Logical  Flow  of  Events 

The  simplified  model  discussed  in  Chapter  IV  provided  a  starting  point  for 
analysis  of  the  range  decision  process.  The  model  provided  a  basic  flow  of  the  decision 
process  followed  by  a  RO  during  a  test  event,  but  lacked  the  detail  to  represent  several 
key  decision  points  such  as  verification  of  initial  conditions  and  determination  of  UUT 
and  test  asset  performance.  The  formal  model  improved  on  this  flow  to  represent  the 
actual  range-event  decision  process. 
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b. 


Decision  Probabilities 


The  simplified  model  had  no  ability  to  represent  the  likelihood  of  the  RO’s 
decision  in  any  situation.  The  use  of  the  Innoslate  tool  provided  the  formal  model  with 
the  flexibility  to  represent  discrete  yes/no  probabilities,  which  in  turn  allowed  for  Monte 
Carlo  simulation  of  possible  decision  flows.  This  was  not  possible  with  the  simplified 
model. 


c.  Frequency  of  Decision  Block  Appearance 

The  simplified  model  discussed  in  Chapter  IV  contained  two  decision  points  as 
compared  to  the  formal  model,  which  has  eleven.  As  discussed  in  Chapter  V,  these 
additions  were  necessary  to  account  for  the  various  decisions  the  RO  actually  makes 
during  a  test  event..  These  additions  allowed  the  model  to  represent  more  closely  the 
multiple  decision  paths  experienced  by  the  RO  during  testing. 

d.  Allowance  for  Aborting  the  Test 

As  discussed  in  Chapter  V,  the  simplified  model  provided  no  option  to  represent 
the  RO  decision  of  aborting  the  test  due  to  technical  or  safety  concerns.  The  formal 
model  provides  four  separate  abort  decision  blocks  that  represent  the  possible  actions 
required  if  the  UUT,  test  assets,  or  the  physical  environment  pose  unacceptable  risks. 

2.  Identification  of  High-Value  Decision  Points 

Chapter  IV  showed  the  frequency  in  which  each  action,  loop,  and  decision  block 
was  active  during  each  of  the  run  increments,  and  the  average  for  all  runs.  Range 
managers  can  conclude  that  because  ROs  encounter  these  decisions  often,  they  have  a 
high  probability  of  affecting  successful  test  outcome  and  therefore  a  high  value.  These 
insights  can  assist  range  managers  in  the  following  areas: 

•  determining  which  decisions  warrant  focused  or  additional  training  when 
qualifying  range  officers,  because  ROs  will  encounter  these  decisions 
often 

•  applying  resources  for  additional  risk  management  in  the  decision  area, 
since  reducing  the  risk  of  an  incorrect  decision  will  reduce  risk  of 
prematurely  aborting  or  adversely  affecting  a  test  run 
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•  improving  range  data  or  situational  awareness  tools  so  that  ROs  can  make 
better  and  more  informed  decisions  in  the  critical  area 

•  comparing  the  high-value  decision  location  in  the  test  sequence  to  the 
probability  of  UUT  vulnerability.  At  certain  intervals  during  the  test,  a 
UUT  may  be  more  vulnerable  to  a  poor  decision  than  during  other 
portions  of  the  test.  Advance  knowledge  that  such  vulnerability  coincides 
with  a  high-value  decision  would  allow  managers  to  better  prepare  for  the 
test  run  and  apply  necessary  mitigations. 

•  determining  personnel  assignments.  Managers  could  analyze  and 
upcoming  test  and  realize  the  risk  of  incorrect  decision-making  is  high, 
and  from  that  determination  assign  a  more  experienced  or  seasoned  RO  to 
minimize  the  risk  of  an  improper  or  unwarranted  decision. 

3.  Usefulness  of  the  Model  to  Range  Managers 

Thesis  assumptions,  research,  development  of  the  formal  model,  and  model  run 
results  were  included  in  a  brief  provided  to  NUWC  Division,  Keyport  range  managers 
and  subject  matter  experts.  Examples  presented  were: 

•  ability  to  represent  key  decisions 

•  ability  to  set  probabilities  for  decision  nodes 

•  formal  model  data  products  showing  frequency  of  various  decision  types 

•  ability  to  conduct  pre-event  planning  through  Innoslate  by  modifying  the 
model  for  different  or  new  types  of  range  events 

•  capability  for  analyzing  decision  points  and  flow  paths  through  Monte 
Carlo  simulation 

•  possibility  of  pre-event  risk  analysis 

•  use  of  the  model  as  a  quantitative  risk  management  tool 

Range  experts  concluded  the  model  has  application  to  their  range  planning 
process.  Discussion  centered  on  how  model  refinements,  especially  quantitative  analysis 
to  generate  accurate  decision  probabilities,  could  greatly  improve  the  model’s  predictive 
accuracy  and  usefulness  in  range  planning.  Range  managers  commented  the  tool  had 
application  to  their  decision  processes  and  that  the  model  provided  a  method  of  analyzing 
the  conduct  of  current  and  potential  range  events. 
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B.  RESOLUTION  OF  THE  THESIS  RESEARCH  QUESTION 

The  thesis  posed  the  question,  can  formally  modeling  the  business  process  of 
conducting  test  range  events  expose  previously  overlooked  ambiguities  and  high-value 
decision  points?  The  formal  model  provided  accurate  representations  of  the  choices 
facing  the  RO,  as  well  as  providing  a  means  to  evaluate  the  relative  worth  of  those 
decisions.  Use  of  the  Innoslate  software  revealed  high  frequency  decision  nodes,  which 
NUWC  range  managers  concluded  were  high  value  decisions.  Thus,  the  formal  model 
provides  a  pre-event  risk  analysis  tool  that  range  managers  can  use  in  both  improving 
existing  types  of  range  events  and  planning  for  first-time  events  testing  new  systems. 
Therefore,  use  of  a  formal  model  representing  a  range  test  event  does  expose  ambiguities 
and  illuminates  high-value  decision  points  in  that  process. 

C.  RECOMMENDATIONS 

The  following  recommendations  resulted  from  the  research  conducted  and  lessons 
learned  from  the  formal  modeling  runs. 

1.  Continue  Model  Improvements 

The  thesis  based  the  formal  model  on  a  simplified  test  range  event.  Range  managers  can 
immediately  improve  on  the  model  by  mapping  out  a  more  accurate  decision  flow  for 
different  types  of  events  to  improve  accuracy.  Likewise,  observations  of  test  events  and 
discussions  with  ROs  and  other  subject  matter  experts  may  reveal  other  decision  points  or 
processes  that  could  improve  model  accuracy.  Additional  recommended  improvements 
include: 

•  Determine  if  an  abort  option  is  required  for  the  Initial  Conditions  Met 
decision  loop..  This  was  not  included  in  the  formal  model  as  the  focus  was 
on  RO  decisions  after  UUT  launch.  For  example,  range  managers  may 
want  to  use  the  model  to  explore  probabilities  of  test  scrubs  due  to  weather 
problems,  and  could  use  this  information  to  inform  potential  customers  of 
the  schedule  risks  if  they  choose  rough  weather  months  for  testing. 

•  Use  data  collection  techniques  to  observe  actual  decisions  and  outcomes  to 
improve  the  accuracy  of  probabilities  assigned  to  the  formal  model 
decision  points.  As  discussed  in  Chapter  V,  higher  accuracy  in  the 
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decision  probabilities  will  lead  to  a  more  accurate  and  predictive  formal 
model. 

2.  Use  as  a  Risk  Management  Tool 

The  thesis  showed  the  formal  model  has  the  flexibility  to  represent  not  only 
existing  events  but  also  could  be  modified  to  represent  future  events  or  potential 
improvements  to  current  test  practices.  For  example: 

•  Range  managers  can  apply  heuristic  experience  from  similar  test  events  to 
modify  the  formal  model  to  represent  untried  events,  and  thus  provide 
quantitative,  pre-event  risk  analysis.  By  using  risk  management  tools  such 
as  event  trees  (Ferdous  et  al.  2011)  and  risk  matrices  (DAU  2017), 
managers  can  identify  potential  issues  by  using  simulation  rather  than 
waiting  for  problems  and  then  applying  corrective  actions  after  the  fact. 
The  range  does  not  currently  use  predictive  modeling  for  risk  analysis  in 
any  fashion  for  new  range  events. 

•  Range  managers  can  also  exploit  the  visual  nature  of  the  formal  model  to 
identify  more  desirable  and  lower-risk  decision  branches  and  then 
determine  the  probabilities  necessary  in  key  decision  blocks  to  accomplish 
those  paths.  Managers  could  then  apply  mitigation  strategies  to  achieve 
those  probabilities,  thus  influencing  the  decision  process  towards  lower 
risk. 

D.  FURTHER  STUDY 

1.  Future  Use  as  of  the  Model  as  an  Acoustic  Test  Range  Planning  Tool 

As  previously  discussed,  the  formal  model  represented  a  simplified  generic  test 
event.  Further  research  is  required  to  improve  the  accuracy  of  the  model  in  representing 
an  event  involving  a  specific  type  of  UUT.  The  author  recommends  further  analysis  to 
determine  whether  the  model  is  sufficiently  accurate  to  replicate  an  actual  test  event. 

2.  Possible  Use  of  the  Model  for  Other  Types  of  Test  Ranges 

The  thesis  showed  the  formal  model  in  generic  terms  to  illustrate  that  the 
approach  and  syntax  of  model  construction  apply  to  many  types  of  decision  processes.  By 
keeping  decision-points  limited  to  yes/no  choices,  complicated  test  processes  received 
sufficient  detail  so  that  a  simplified  test  event  process  was  recognizable.  Therefore,  this 
modeling  approach  is  not  limited  to  acoustic  testing  alone  and  has  application  to  other 
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types  of  test  ranges,  and  has  application  to  any  decision  process  capable  of  simplification 
to  a  yes/no  decision  flow.  While  this  limitation  does  not  support  all  decision  processes, 
findings  listed  in  this  thesis  indicate  useful  analysis  and  insight  are  possible  with  the 
formal  model  approach. 
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APPENDIX  A.  FORMAL  MODEL  BLOCK  IDENTIFICATION, 
TIMING,  PROBABILITY  VALUES 


This  appendix  provides  details  used  to  define  and  set  parameters  used  to  populate 
the  formal  model  activity  blocks  in  Innoslate. 


FORMAL  MODEL  BLOCK  IDENTIFICATION  &  PARAMETERS  | 

Block 

ID 

Innoslate 

ID 

Block  Label 

Previous  Block 

Activity 

Function 

Description 

Duration 

(Minutes) 

Script 

Probability  Values 

_ m _ 

Yes 

No 

1 

e7NlSH 

Verify  Initial  Conditions 

None 

Action 

Represent  overall  activity  of  establishing 

360 

None 

N/A 

N/A 

2 

e20PAl 

Required  Initial  Conditions 
Met? 

1 

OR 

Decision  node 

RO  verification  that  initial  conditions 
support  UUT  launch 

30 

Probability 

10 

90 

3 

e3N61 

Proceed  to  UUT  launch 

2 

Action 

Action  if  outcome  to  2  is  YES 

Account  for  time  necessary  to  provide 
directioas  to  launch  UUT 

20 

None 

N/A 

N/A 

D 

e6QR9J 

Establish  Proper  Conditioas 

2 

Action 

Action  if  outcome  to  2  is  NO 

Account  for  time  necessary  to  provide 
directioas  to  launch  UUT 

60 

None 

N/A 

N/A 

5 

e7NlSJ 

Conduct  Test 

1 

Action 

Overall  activity  for  test  conduct 

60 

None 

N/A 

N/A 

6 

e8KW41 

Begin  Test  Event  (Launch) 

5 

Action 

Provide  starting  action 

Specific  action  in  the  formal  model  to 
represent  test  event  start 

5 

None 

N/A 

N/A 

D 

e20CHH 

Monitor  Test  Performance 
Until  Test  Objectives 
Complete 

6 

LOOP 

Allow  recursive  monitoring  of  test 
sequencing 

Allows  model  to  represent  RO's 
monitoring  to  the  test  event  as  the  test 
sequence  occurs 

30 

Continues  until 
encountering  a 
FALSE  value 

N/A 

N/A 

8 

e3WGC2 

Test  Sequence  Objectives 
Met? 

7 

OR 

Decision  node 

2 

Probability 

20 

80 

D 

e8M5WH 

Continue  Monitoring  ( 1 ) 

8 

Action 

Action  if  outcome  to  8  is  YES 

Represents  duration  of  monitoring 
while  test  is  progressing  normally 

3 

None 

N/A 

N/A 

10 

e3XQEl 

UUT  Performance  or 

Environmental/Test  Asset 

Problem? 

8 

OR 

Action  if  outcome  to  8  is  NO 

■ 

Probability 

40 

60 

11 

e7NBJ  1 

Evaluate  UUT  Performance 

10 

Action 

Action  if  outcome  of  10  is  "UUT 

Problem" 

Decomposes  to  activities  supporting 
evaluation  of  UUT  performance 

5 

None 

N/A 

N/A 

12 

el  1J71 

Evaluate  Env.  &  Test 
Support  Asset  Performance 

10 

Action 

Action  if  outcome  of  10  is 
"Environment/Test  Asset  Problem" 

Decomposes  to  activities  supporting 

evaluation  of  Environmental/Test 
Support  Asset  performance 

■ 

None 

N/A 

N/A 

13 

e3WT4H 

UUT  Performance  as 
Expected? 

11 

OR 

Decision  node 

RO  decision  on  whether  UUT 
performance  is  as  expected  based  on 
the  run  plan  and  RO  experience 

1 

Probability 

90 

10 

14 

e5T5 1 1 

Abort  Required?  (1) 

13 

OR 

Action  if  outcome  of  1 3  is  YES 

If  the  RO  observes  a  problem  but  the 

UUT  is  performing  normally,  he  must 
decide  if  a  test  abort  is  required 

■ 

Probability 

5 

95 

15 

Abort  Test  (1) 

14 

Captures  duration  if  RO  aborts  test 

3 

None 

16 

e2274H 

Continue  Monitoring  (2) 

14 

Action 

Action  if  outcome  of  14  Is  NO 

Captures  duration  if  RO  continues  test 

3 

None 

N/A 

N/A 

17 

e3X3Xl 

Take  Available  Action  (1) 

11 

Action 

Capture  duration  of  corrective  actions 

Represents  actions  to  correct  noted 
problems  during  tests 

2 

None 

N/A 

N/A 

18 

e6RBTH 

Action  Effective?  (1) 

17 

OR 

Decision  node 

RO  evaluation  of  corrective  action 

effectiveness 

1 

Probability 

20 

80 

19 

e8N361 

Continue  Monitoring  (3) 

18 

Action 

Action  if  outcome  of  1 8  Is  YES 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 

20 

e5SHG2 

Abort  Required?  (2) 

18 

OR 

Action  if  outcome  of  1 8  is  NO 

If  action  taken  Is  ineffective,  RO  must 
decide  if  test  should  continue 

1 

Probability 

10 

90 

21 

Abort  Test  (2) 

Duration  of  actions  for  test  abort 

3 

None 

22 

e7NZ31 

20 

Action 

Action  if  outcome  of  20  Is  NO 

Even  if  corrective  action  was 
ineffective,  the  RO  can  choose  to 
continue  the  test 

None 

N/A 

N/A 

23 

e5SHGl 

Problem  with  Environment 
or  Test  Support  Assets? 

12 

OR 

Decision  node 

RO  decides  if  test  assets  or 
environment  (e.g.  weather,  sea  state) 
are  a  problem 

Probability 

10 

90 

24 

Take  Available  Action  (2) 

23 

Implement  corrective  actions 

5 

None 

25 

e2QWH 

Action  Effective?  (2) 

24 

OR 

Decision  node 

warn*,  ---.vj; 

1 

Probability 

60 

40 

26 

e8N361 

Continue  Monitoring  (4) 

25 

Action 

Action  if  outcome  of  25  is  YES 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

27 

e8KJBJ 

Abort  Required?  (3) 

25 

OR 

Action  if  outcome  of  25  Is  NO 

1 

10 

90 

28 

e2XZTl 

Abort  Test  (3) 

27 

Action 

Action  if  outcome  of  27  is  YES 

Duration  of  actions  for  test  abort 

3 

None 

N/A 

N/A 

29 

e4VY7H 

Continue  Monitoring  (5) 

25 

Action 

Action  if  outcome  of  27  is  NO 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 

30 

e21KKH 

Abort  Required?  (4) 

23 

OR 

Action  if  outcome  of  23  is  NO 

If  action  taken  is  ineffective,  RO  mast 
decide  if  test  should  continue 

1 

Probability 

10 

90 

31 

Abort  Test  (4) 

Action 

Action  if  outcome  of  30  Is  YES 

Duration  of  actions  for  test  abort 

3 

None 

N/A 

N/A 

32 

e3BDH 

Continue  Monitoring  (6) 

30 

Action 

Action  if  outcome  of  30  is  NO 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 
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APPENDIX  B.  FORMAL  MODEL  ADJUSTMENTS  FOR 
SENSITIVITY  ANALYSIS 


This  appendix  shows  the  parameter  changes  to  two  activity  blocks  in  the  formal 
model.  These  changes  provided  a  simple  method  to  observe  the  sensitivity  of  the  model 
to  changes  in  decision  probability. 


FORMAL  MODEL  BLOCK  IDENTIFICATION  &  PARAMETERS 

Block 

ID 

Innoslate 

ID 

Block  Label 

Previous  Block 

Activity 

Function 

Description 

Duration 

(Minutes) 

Script 

Probability  Values 

m 

Yes 

No 

1 

e7NlSH 

Verily  Initial  Conditions 

None 

Action 

Represent  overall  activity  of  establishing 
necessary  conditions  for  test 

Provides  duration  for  test  event 
preparations 

360 

None 

N/A 

N/A 

2 

e20PAl 

Required  Initial  Conditions 
Met? 

1 

OR 

Decision  node 

RO  verification  that  initial  conditions 
support  UUT  launch 

30 

Probability 

10 

90 

3 

e3N61 

Proceed  to  UUT  launch 

2 

Action 

Action  if  outcome  to  2  is  YES 

Account  for  time  necessary  to  provide 
directions  to  launch  UUT 

20 

None 

N/A 

N/A 

4 

e6QR9J 

Establish  Proper  Conditions 

2 

Action 

Action  if  outcome  to  2  is  NO 

Account  for  time  necessary  to  provide 
directions  to  launch  UUT 

60 

None 

N/A 

N/A 

5 

e7NlSJ 

Conduct  Test 

1 

Action 

Overall  activity  for  test  conduct 

Decomposes  to  varions  activities  that 
represent  test  conduct 

60 

None 

N/A 

N/A 

6 

e8KW41 

Begin  Test  Event  (Launch) 

5 

Action 

Provide  starting  action 

Specific  action  in  the  formal  model  to 
represent  test  event  start 

5 

None 

N/A 

N/A 

7 

e20CHH 

Monitor  Test  Performance 
Until  Test  Objectives 
Complete 

6 

LOOP 

Allow  recursive  monitoring  of  test 
sequencing 

Allows  model  to  represent  RO's 
monitoring  to  the  test  event  as  the  test 
sequence  occurs 

30 

Continues  until 
encountering  a 
FALSE  value 

N/A 

N/A 

8 

e3WGC2 

Test  Sequence  Objectives 
Met? 

7 

OR 

Decision  node 

RO  verification  that  the  test  is 
progressing  per  the  run  plan 

2 

Probability 

20 

80 

9 

e8M5WH 

Continue  Monitoring  (1) 

8 

Action 

Action  if  outcome  to  8  is  YES 

Represents  duration  of  monitoring 
while  test  is  progressing  normally 

3 

None 

N/A 

N/A 

10 

e3XQEl 

UUT  Performance  or 

Environmental/Test  Asset 

Problem? 

8 

OR 

Action  if  outcome  to  8  is  NO 

RO  decision  on  whether  problem  is 
UUT  or  Environment/Test  Asset 

related 

5 

Probability 

40 

60 

11 

e7NBJl 

Evaluate  UUT  Performance 

10 

Action 

Action  if  outcome  of  10  is  'XJUT 

Problem" 

Decomposes  to  activities  supporting 
evaluation  of  UUT  performance 

5 

None 

N/A 

N/A 

12 

el  1171 

Evaluate  Env.  &  Test 
Support  Asset  Performance 

10 

Action 

Action  if  outcome  of  10  is 

'Environment/Test  Asset  Problem" 

Decomposes  to  activities  supporting 
evaluation  of  Environmental/Test 
Support  Asset  performance 

5 

None 

N/A 

N/A 

13 

e3WT4H 

UUT  Performance  as 
Expected? 

11 

OR 

Decision  node 

RO  decision  on  whether  UUT 
performance  is  as  expected  based  on 
the  run  plan  and  RO  experience 

1 

Probability 

90 

10 

14 

e5T511 

Abort  Required?  (1) 

13 

OR 

Action  if  outcome  of  1 3  is  YES 

If  the  RO  observes  a  problem  but  the 
UUT  is  performing  normally,  he  must 
decide  if  a  test  abort  is  required 
regardless 

1 

Probability 

5 

95 

15 

ellVZH 

Abort  Test  (1) 

14 

Action 

Action  if  outcome  of  14  is  YES 

Captures  duration  if  RO  aborts  test 

3 

None 

N/A 

N/A 

16 

e2274H 

Continue  Monitoring  (2) 

14 

Action 

Action  if  outcome  of  14  is  NO 

Captures  duration  if  RO  continues  test 

3 

None 

N/A 

N/A 

17 

e3X3Xl 

Take  Available  Action  (1) 

11 

Action 

Capture  duration  of  corrective  actions 

Represents  actions  to  correct  noted 
problems  during  tests 

2 

None 

N/A 

N/A 

18 

e6RBTH 

Action  Effective?  (1) 

17 

OR 

Decision  node 

RO  evaluation  of  corrective  action 

effectiveness 

1 

Probability 

20 

80 

19 

e8N361 

Continue  Monitoring  (3) 

18 

Action 

Action  if  outcome  of  1 8  is  YES 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 

20 

e5SHG2 

Abort  Required?  (2) 

18 

OR 

Action  if  outcome  of  1 8  is  NO 

If  action  taken  is  ineffective,  RO  must 
decide  if  test  should  continue 

1 

Probability 

10 

90 

21 

e3XDNH 

Abort  Test  (2) 

20 

Action 

Action  if  outcome  of  20  is  YES 

Duration  of  actions  for  test  abort 

3 

None 

N/A 

N/A 

22 

e7NZ31 

Allow  Test  to  Continue 

20 

Action 

Action  if  outcome  of  20  is  NO 

Even  if  corrective  action  was 
ineffective,  the  RO  can  choose  to 
continue  the  test 

3 

None 

N/A 

N/A 

23 

e5SHGl 

Problem  with  Environment 
or  Test  Support  Assets? 

12 

OR 

Decision  node 

RO  decides  if  test  assets  or 
environment  (e.g.  weather,  sea  state) 
are  a  problem 

2 

Probability 

10 

90 

24 

e3X3Xl 

Take  Available  Action  (2) 

23 

Action 

Action  if  outcome  of  23  is  YES 

Implement  corrective  actions 

5 

None 

N/A 

N/A 

25 

e2QWH 

Action  Effective?  (2) 

24 

OR 

Decision  node 

RO  evaluation  of  corrective  action 

effectiveness 

1 

Probability 

40 

60 

26 

e8N361 

Continue  Monitoring  (4) 

25 

Action 

Action  if  outcome  of  25  is  YES 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 

27 

e8KJBJ 

Abort  Required?  (3) 

25 

OR 

Action  if  outcome  of  25  is  NO 

If  action  taken  is  ineffective,  RO  must 
decide  if  test  should  continue 

1 

Probability 

30 

70 

28 

e2XZTl 

Abort  Test  (3) 

27 

Action 

Action  if  outcome  of  27  is  YES 

Duration  of  actions  for  test  abort 

3 

None 

N/A 

N/A 

29 

e4VY7H 

Continue  Monitoring  (5) 

25 

Action 

Action  if  outcome  of  27  is  NO 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 

30 

e21KKH 

Abort  Required?  (4) 

23 

OR 

Action  if  outcome  of  23  is  NO 

If  action  taken  is  ineffective,  RO  must 
decide  if  test  should  continue 

1 

Probability 

10 

90 

31 

e6S941 

Abort  Test  (4) 

30 

Action 

Action  if  outcome  of  30  is  YES 

Duration  of  actions  for  test  abort 

3 

None 

N/A 

N/A 

32 

e3BDH 

Continue  Monitoring  (6) 

30 

Action 

Action  if  outcome  of  30  is  NO 

Represents  duration  of  monitoring  if 

RO  decides  to  test  can  continue 

3 

None 

N/A 

N/A 
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